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GENERIC  DISTRIBUTED  SYSTEM  MODEL 


Summary 


This  report  identifies  various  issues  associated  with  the  design  of  Distributed  Systems 
and  presents  an  overview  of  modeling.  It  is  felt  that  the  tools  are  primarily  oriented 
towards  development  of  software  to  run  on  distributed  systems  rather  than  the  design 
tools  for  distributed  systems.  During  my  four  months’  of  stay  at  AIRMICS,  information 
about  some  design  tools  were  acquired.  But  additional  efforts  are  needed  to  find  out  other 
modeling  efforts  being  pursued  by  various  other  funding  agencies  and  research  organiza¬ 
tions  so  that  any  replication  of  efforts  could  be  avoided  and  the  problem  of  Distributed 
System  Design  for  a  given  requirements  and  specifications  could  be  addressed  effectively. 
Future  directions  and  recommendations  have  also  been  included  as  a  part  of  this  report. 


Introduction 


The  term, distributed  system,  applies  to  a  very  generic  class  of  decentralized  systems. 
It  encompasses  all  systems  having  geographically  distributed  multiple  processors  with 
some  kind  of  coordination  between  them.  Usually,  people  confuse  a  simple  networking  of 
microcomputers  or  work-stations  with  a  distributed  system  and  a  clear  distinction 
between  the  two  needs  to  be  made.  -What  is  expected  in  a  distributed  system  is  an 
appropriate  coordination  of  data  and  control  among  various  functional  units.  In  a 
networked  system,  all  the  computers  are  autonomous  and  each  one  works  independently 
of  others,  even  though  messages  could  be  transferred  between  themv  The  name  distri¬ 
buted  system  also  implies  that  the  resources  (including  computers)  are  dispersed  physi¬ 
cally,  with  one  or  many  locations  acting  as  the  source  of  data  and  others  taking  care  of 
processing  the  data  and  making  decisions.  The  area  of  distributed  processing  is  relatively 
new  and  increased  emphasis  is  needed  to  obtain  major  results.  There  are  numerous  open 
questions  and  all  of  these  need  to  be  addressed  carefully  to  provide  results  of  substantial 
importance.  The  complication  in  such  efforts  arises  from  various  associated  factors  and 
some  of  them  are  as  follows: 

i.  How  many  and  what  type  of  resources  are  required  in  a  heterogeneous  environment? 

ii.  What  type  of  data  is  anticipated? 

iii.  What  are  the  user  defined  requirements  that  could  adequately  represent  the  real-life 
situation? 

iv.  What  is  the  best  way  of  translating  users’  requirements  into  system  level  require¬ 
ments? 

v.  What  are  the  appropriate  ways  of  system  modeling? 

vi.  What  techniques  ought  to  be  used  in  partitioning  the  system  model? 

vii.  What  are  the  target  system  architectures  and  network  topologies? 

viii.  What  are  the  ways  to  incorporate  the  multi-media  networks? 

ix.  How  to  take  into  account  the  reliability,  graceful  degradation  and  recovery  aspects 
concurrently? 

x.  What  strategies  ought  to  be  used  in  allocating  files  and  system  software  to  various 
nodes  of  the  system? 

xi.  What  alternate  nodal  design  ought  to  be  considered? 

xii.  What  are  the  essential  performance  parameters? 

It  may  be  noted  that  the  distributed  system  issues  have  been  addressed  in  a  piece¬ 
meal  manner  and  much  attention  have  been  paid  to  the  modeling  of  software  for  distri¬ 
buted  system  applications.  There  has  been  very  little  work  on  the  modeling  of  the  overall 
distributed  system  and  an  overview  of  existing  work  is  included  in  the  next  section. 
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Overview  of  Existing  Work 


The  design  of  a  distributed  system  for  a  given  class  of  applications  is  a  very  complex 
problem  and  has  been  of  constant  interest  to  the  research  and  development  community. 
Some  of  the  analysis  techniques  have  been  covered  in  [CHA80]  while  an  overview  of  distri¬ 
buted  computer  systems  has  been  given  in  [STA84J.  Gonzalez  and  Jordan  [GON80]  were 
the  first  ones  to  come  up  with  the  framework  for  quantitative  evaluation  of  distributed 
computer  systems  and  some  details  are  given  in  the  next  section  [MCA87).  The  four 
major  areas  covered  in  the  literature  are  the  network  architectures,  the  resource  alloca¬ 
tion,  the  software  design  issues  and  the  language  selection.  Networks  (PRA85j  suitable 
for  both  parallel  and  distributed  processing  have  been  introduced  in  the  literature.  A 
comparative  study  [AGR85]  of  many  such  networks  have  been  performed  in  terms  of  vari¬ 
ous  architectural  characteristics  such  as  average  distance,  maximum  distance  (diameter) 
number  of  alternate  paths,  ease  of  expansion,  simplicity  of  routing,  and  number  of 
input/output  ports  per  node.  The  effectiveness  of  these  network  configurations  from  an 
algorithmic  point  of  view  has  also  been  determined  [AGR85]  when  some  of  the  actual 
algorithms  are  mapped  onto  various  architectures.  Infact,  this  comparative  study 
[AGR85]  led  to  the  development  of  the  B-HIVE  [AGR88],  a  24-node  multicomputer  pro¬ 
ject  at  the  North  Carolina  State  University  which  is  nearing  completition  and  will  be  used 
for  parallel  and  distributed  processing  experimentation. 

The  resource  allocation  (sometimes  also  called  mapping  in  the  context  of  parallel 
processing)  problem  has  been  addressed  by  various  researchers  [PAT84,  COR87,  KAL87, 
TAN85,  WAN85,  YAS87,  RAM83,  GAI87,  HOU87).  The  basic  idea  in  all  these  efforts  is 
to  have  models  of  the  applications  and  allocate  various  parts  of  the  models  to  different 
processors  or  nodes  of  the  system.  The  allocation  is  done  either  statically  or  pseudo- 
dynamically  and  the  usual  performance  parameters  to  be  optimized  are  the  turn-around 
time,  the  speed  up,  the  processor  utilization  and  the  channel  utilization.  These  parame¬ 
ters  are  similar  when  applied  to  a  distributed  system  and  a  parallel  system.  Most  of  the 
researchers  consider  all  the  tasks  independent  of  others  and  are  usually  allocated  to 
reduce  the  net  turn-around  time.  For  parallel  system,  the  precedence  relationship  (or 
dependence)  is  also  taken  in  to  account  while  doing  the  mapping.  The  dependency  rela¬ 
tions  are  incorporated  in  some  distributed  systems  [CAT87]  in  the  form  of  data-flow 
graphs  when  employed  for  real-time  applications.  This  needs  to  be  done  for  any  applica¬ 
tion  as  long  as  some  coordination  is  desirable  and  in  future,  there  will  be  increased 
emphasis  on  this  aspect.  Recent  results  on  allocation  [LEU87,  CAS88,  CHI88)  emphasize 
the  need  for  minimization  of  communication  overhead  by  either  overlapping  the  com¬ 
munication  with  computation  or  appropriately  managing  the  resources.  The  static  map¬ 
ping  with  no  run-time  overhead,  is  an  approximation  of  the  actual  requirements;  while 
the  dynamic  scheduling  done  at  run  time,  in  general,  requires  prohibitively  large  over¬ 
head.  A  good  compromise  is  yet  to  be  found.  In  a  similar  way,  there  are  many  open  ques¬ 
tions  and  much  more  research  is  needed  in  this  area. 
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The  software  issues  have  been  the  bottleneck  even  for  a  uniprocessor  system  and 
software  modeling  for  a  distributed  system  has  been  a  lively  subject  and  poses  new  chal- 
langes.  Some  of  the  noteworthy  approaches  include  [SHA87,  IND86,  CHE80,  RAM85] 
while  the  recent  spiral  model  [BOE86]  advocates  the  adoption  of  successive  enhancements 
untill  the  specifications  are  satisfied.  In  terms  of  actual  coding,  relative  advantages  and 
disadvantages  of  various  languages  have  been  described  in  [FOR86].  The  use  of  commun¬ 
icating  sequential  processes  for  distributed  system  software  has  been  demonstrated  in 
(HAN87,  RAV87],  The  AIRMICS  is  in  process  of  acquiring  the  Joyce  environment  from 
Syracuse  University  and  this  could  prove  to  be  very  useful.  It  may  be  noted  that  all  these 
efforts  have  been  directed  towards  development  of  efficient  software  to  run  on  a  distri¬ 
buted  system,  rather  than  a  tool  for  distributed  system  design.  In  January  1988,  when  I 
joined  AIRMICS,  the  only  design  tool  available  in-house  was  the  McCable  Software  pack¬ 
age.  It  took  a  while  to  set  up  an  appropriate  system  configuration  so  that  its  capabilities 
could  be  tested.  During  my  four  months'  stay  at  AIRMICS,  I  was  able  to  find  out  more 
about  some  other  design  tool  efforts  being  pursued.  In  April  1988,  we  were  able  to  acquire 
the  executable  code  for  the  TRW  DCDS  software  package  from  the  US  Army  Strategic 
Defense  Command  at  Huntsville.  But,  because  of  numerous  reasons,  we  were  not  able  to 
install  this  DCDS  package  neither  at  AIRMICS  nor  at  the  Georgia  Tech  Computing 
Center. 

Other  modeling  efforts  that  we  came  accross  have  been  associated  with  the  USA 
ISEC  Information  Software  Support  Command  (ISSC)  Quality  Assurance  Directorate 
(QAD)  and  the  USAF  Rome  Air  Development  Center  (RADC).  The  RADC  work 
(RAD86)  written  in  SYMSCRIPT,  has  been  basically  applicable  for  the  network  modeling 
and  will  be  released  in  July  1989  after  appropriate  update.  The  QAD  has  been  support¬ 
ing  queueing  delay  computation  work  for  IBM  machines  and  is  currently  seeking  help  in 
modeling  Sperry  5000  systems.  We  tried  our  best  to  get  their  reports,  but  as  of  this  writ¬ 
ing,  we  have  not  received  any  documents. 
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McCabe  Design  Tool 


The  McCabe  Design  Tool  [MCC87]  has  been  developed  under  the  sponsorship  of 
AIRMICS.  The  basic  idea  is  to  have  a  tool  for  distributed  system  design  with  a  capability 
of  varying  different  parameters  and  observing  their  impact  on  various  quality  attributes 
such  as: 

-Availability, 

-Efficiency, 

-Flexibility, 

-Integrity, 

-Maintainability, 

-Reliability, 

-Survivability. 

These  parameters  are  good  to  obtain  a  reasonable  idea  of  the  design  adequacies.  But 
a  careful  execution  of  the  software  would  reveal  that  the  designer  is  expected  to  provide  a 
lot  of  information  while  no  optimization  technique  has  been  used.  For  example,  the 
designer  needs  to  specify  all  the  requirements  as  follows  (Fig.  1): 

-The  no.  of  external  request  for  processing  per  day. 

-The  physical  distance  between  various  places  where  queries  are  generated. 

-The  type  of  processing  required  in  terms  of  function  (or  process). 

-The  time  required  by  each  process. 

-The  probability  of  one  process  utilizing  another  process. 

-Placement  restrictions  on  some  external  requests  or  for  some  processes. 

-Any  other  restrictions  or  requirements  such  as  response  time,  etc. 

Once  these  have  been  supplied  in  the  form  of  the  model  (see  Fig.  2),  and  the  probable 
physical  locations  of  each  process  have  been  given,  then  the  software  package  computes 
various  attributes  (shown  in  Table  I  and  Table  II).  At  this  point,  it  is  possible  to  change 
the  location  of  each  process,  the  distance  between  the  query  sources,  or  the  weights 
assigned  to  various  attributes  and  the  impact  on  the  net  system  performance  could  be 
observed.  In  this  way,  a  lot  of  information  has  to  be  provided  by  the  designer  and  there  is 
no  provision  for  parameter  optimization.  All  changes  have  to  be  done  manually  and  no 
decision  could  be  made  about  the  type  of  processing  nodes  to  be  used  or  the  type  of  chan¬ 
nels  and  their  capacity  to  be  employed.  But,  it  could  be  said  to  be  a  mere  starting  point 
and  substantial  efforts  are  needed  to  make  it  comprehensive  for  today’s  technology. 
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INFORMATION  PERTAINING  TO  LOCATIONS  AND  POLICY  CAN  BE  COMBINED 
WITH  THE  INITIAL  DIAGRAM  TO  PRODUCE  A  PARTITIONED  D  f  0  .  THE  BOLD 
LINES  REPRESENT  SERVICES  WHICH  WERE  PLACED  BECAUSE  Of  MANDATORY 
POLICY  CONSTRAINTS.  THE  REMAINING  PROCESSES  WERE  LOCATED 
ARBITRARILY  FOR  THIS  BASELINE  ALLOCATION. 


Fig.  1. POLICY  DICTATES  THAT  THE  APPLICATION  HEADQUARTERS  «L! 
MAINTAIN  THE  PRESCRIBED  LOAD  LIST  (PLO)  AND  THE  POST  RECEIPTS 
DEPARTMENT  .  IN  ADDITION,  REQUISITION  SITES  MUST  BE  LOCATED  AT 
FT.  3ELVOIR  AND  FT.  ORD  SO  THAT  REQUISITIONS  CAN  k  MADE 
DIRECTLY. 


COMPLETE  DFD  CREATED  USING  DESIGN  AID,  APPEARS  BELOW. 
THE  DIAGRAM  IS  APPENDED  WITH  THE  SPECIAL  SYMBOLS  WHICH  CONTAIN 
THE  SYSTEM  PARAMETERS:  LOCATION,  NAME,  UTILIZATION,  EXTERNAL 
ENTRIES,  DATA  FLOW  PROBABILITIES. 
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Table  I.  PERFORMANCE  ANALYSIS  RESULTS 
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TRW  Design  Tool 


The  TRW  Distributed  Computer  System  Design  (DCDS)  has  been  an  outcome  of  13 
years  of  efforts  by  the  TRW  and  has  been  sponsored  by  the  U.S.  Army  Strategic  Defense 
Command  at  Huntsville,  Alabama.  The  design  tools  seem  to  be  reasonably  mature  even 
though  the  emphasis  seems  to  be  in  the  software  design  methodology  area.  We  were  not 
able  to  do  enough  experimentation  because  of  the  timing  constraints.  The  non-availability 
of  any  systematic  guideline  or  tutorial  for  using  the  software  package  delayed  our  under¬ 
standing  and  assessment  of  the  package.  A  thorough  experimentation  is  desirable  to 
appreciate  the  capabilities  of  the  tool.  This  would  eliminate  any  duplication  of  identical 
work.  The  DCDS  consists  of  five  different  methodologies  and  are  shown  in  Fig.  3.  For 
each  one  of  them,  a  separate  but  similar  language  has  been  used  and  is  shown  in  Table  III. 

Table  III  Different  Methodologies  and  Corresponding  Languages 


No.  Method 


Language 


1.  Systems  Requirements 

Engineering  Methodology 
(SYSREM) 


System  Specification 
Language 
(SSL) 


2.  Software  Requirements 
Engineering  Methodology 
(SREM) 


Requirements  Statement 
Language 
(RSL) 


3.  Distributed  Design  Methodology 
(DDM) 


Distributed  Design  Language 
(DDL) 


4. 


Module  Developmen  Methodology  Module  Development  Language 
(MDM)  (MDL) 


5.  Test  Support  Methodology 
(TSM) 


Test  Support  Language 

(TSL) 
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The  selection  of  (he  design  methodologies  totally  depends  on  the  type  of  require¬ 
ments  given  to  the  designer.  Each  of  these  methodologies  have  several  phases  and  it  is 
possible  to  see  the  relevance  between  different  corresponding  steps  as  well  as  various  attri¬ 
butes.  The  specifications  need  to  be  defined  by  the  designer  and  it  seems  to  be  more  or  less 
static  in  nature.  Any  changes  in  specifications  necessitate  rerun  of  the  complete  package. 
As  we  have  not  run  an  actual  example,  it  is  difficult  to  comment  on  the  performance 
parameters  to  be  produced  by  the  DCDS.  However,  it  seems  that  some  attributes  impor¬ 
tant  to  the  Army  such  as  actual  Throughput,  Node  Utilization,  Channel  Utilization,  Reli¬ 
ability  and  Graceful  Degradation  could  be  missing. 
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The  DCDS  Components 


Conclusions  and  Recommendations 


The  Distributed  Computer  System  Design  is  still  in  the  infancy  stage  and  prolonged 
efforts  are  needed  to  gain  adequate  insight  to  various  involved  issues.  To  start  with,  it  is 
important  to  have  some  simple  design  tools  which  could  be  easily  migrated  to  and/or 
adopted  at  AIRMICS.  With  the  experience  I  had  at  AIRMICS,  it  could  be  easily  said  that 
the  existing  McCabe  Software  package  currently  operational  at  AIRMICS,  has  very  lim¬ 
ited  use  and  it  is  inappropriate  for  possible  extensions  both  from  the  economical  and  the 
technical  aspects.  The  DCDS  software  package  from  the  TRW  received  towards  the  end 
of  my  stay  at  AIRMICS,  look  promising  and  further  detailed  experimentation  is  needed  to 
evaluate  and  establish  its  capabilities  and  limitations.  There  are  two  major  issues  to  be 
addressed.  The  first  is  the  establishment  of  an  appropriate  hardware  configuration  so  that 
the  TRW  DCDS  software  could  be  run  successfully.  This  might  require  purchase  and 
installation  of  additional  equipment  either  at  the  AIRMICS  or  at  the  Georgia  Tech  Com¬ 
puting  Center.  The  second  issue  is  to  deal  with  the  source  code  for  the  DCDS  software 
package.  It  is  mandatory  to  have  the  source  code  if  any  modifications,  changes  or  aug¬ 
mentations  in  existing  DCDS  software  are  to  be  done  for  further  enhancements.  Hence, 
efforts  must  also  be  made  to  acquire  the  source  code  either  from  the  TRW  or  from  the  US 
ASDC.  Additional  efforts  are  needed  to  find  out  the  design  efforts  being  supported  by 
other  Government  funding  agencies  as  well  as  in-house  research  being  carried  out  by  vari¬ 
ous  institutions  and  research  organizations. 

It  seems  to  me  that  many  design  and  development  issues  of  current  interest  have 
either  been  partially  covered  or  have  not  been  addressed  at  all  by  the  TRW  software 
package  while  implementing  the  distributed  system  design.  This  may  due  to  the  fact  that 
many  newer  techniques  have  now  been  introduced  and  due  to  advances  in  VLSI  design 
technology,  the  issues  associated  with  DCDS  design,  are  different  than  the  time  the  DCDS 
work  at  TRW  started.  A  detailed  study  of  other  design  efforts  would  also  be  helpful  in 
identifying  various  issues  and  would  provide  an  insight  to  several  alterative  strategies  for 
handling  the  same  problems.  In  any  case,  some  of  the  obvious  but  nontrivial  issues  could 
be  given  as  follows: 

1.  How  do  you  test  the  adequacy  of  transformation  from  designers’  to  systems’  require¬ 
ments  [ARM87]? 

2.  What  partitioning  strategy  should  be  used  in  assigning  the  locations  of  various  func¬ 
tional  units  and  what  functions  to  be  assigned  solely  to  the  hardware? 

3.  Which  network  configurations  are  appropriate  for  Distributed  System  Design? 
Whether  a  simple  ring  structure  is  adequate  or  one  needs  to  look  into  2- in  2-out  net¬ 
works  [CHU88]  or  other  complex  networks  from  the  throughput  and  reliability 
viewpoint  [KUM88]? 

4.  What  file  allocation  scheme  ought  to  be  used  from  the  reliability  or  graceful  degrada¬ 
tion  view  point  [MAH88]? 
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5.  What  mechanism  or  protocol  ought  to  he  used  to  reduce  the  communicat  ion  over¬ 
head  [CH088]? 

6.  What  modifications/changes  need  to  be  done  in  existing  modeling  and  simulation 
software?  How  do  you  check  the  correctness  [ARM87]? 

7.  What  type  of  prototype  testbed  is  needed  to  validate  the  results? 

It  may  be  noted  that  this  is  not  a  comprehensive  list.  AIRMICS  should  provide  sub¬ 
stantial  support  in  setting  up  some  kind  of  a  test  bed  for  exhaustive  experimentation  and 
the  aforementioned  issues,  in  some  prioritized  manner,  ought  to  be  addressed  immedi¬ 
ately. 

Finally,  it  would  be  useful  to  keep  an  eye  on  what  other  groups  have  been  doing,  as 
there  is  no  point  in  reinventing  the  wheel.  It  would  be  adviseable  to  keep  monitoring  what 
different  institutions  are  doing  and  then  look  in  to  appropriate  portions/modules  for 
future  intregration. 
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